5 research outputs found
BLP 2023 Task 2: Sentiment Analysis
We present an overview of the BLP Sentiment Shared Task, organized as part of
the inaugural BLP 2023 workshop, co-located with EMNLP 2023. The task is
defined as the detection of sentiment in a given piece of social media text.
This task attracted interest from 71 participants, among whom 29 and 30 teams
submitted systems during the development and evaluation phases, respectively.
In total, participants submitted 597 runs. However, a total of 15 teams
submitted system description papers. The range of approaches in the submitted
systems spans from classical machine learning models, fine-tuning pre-trained
models, to leveraging Large Language Model (LLMs) in zero- and few-shot
settings. In this paper, we provide a detailed account of the task setup,
including dataset development and evaluation setup. Additionally, we provide a
brief overview of the systems submitted by the participants. All datasets and
evaluation scripts from the shared task have been made publicly available for
the research community, to foster further research in this domainComment: Accepted in BLP Workshop at EMNLP-2
Z-Index at CheckThat! Lab 2022: Check-Worthiness Identification on Tweet Text
The wide use of social media and digital technologies facilitates sharing
various news and information about events and activities. Despite sharing
positive information misleading and false information is also spreading on
social media. There have been efforts in identifying such misleading
information both manually by human experts and automatic tools. Manual effort
does not scale well due to the high volume of information, containing factual
claims, are appearing online. Therefore, automatically identifying check-worthy
claims can be very useful for human experts. In this study, we describe our
participation in Subtask-1A: Check-worthiness of tweets (English, Dutch and
Spanish) of CheckThat! lab at CLEF 2022. We performed standard preprocessing
steps and applied different models to identify whether a given text is worthy
of fact checking or not. We use the oversampling technique to balance the
dataset and applied SVM and Random Forest (RF) with TF-IDF representations. We
also used BERT multilingual (BERT-m) and XLM-RoBERTa-base pre-trained models
for the experiments. We used BERT-m for the official submissions and our
systems ranked as 3rd, 5th, and 12th in Spanish, Dutch, and English,
respectively. In further experiments, our evaluation shows that transformer
models (BERT-m and XLM-RoBERTa-base) outperform the SVM and RF in Dutch and
English languages where a different scenario is observed for Spanish.Comment: Accepted in CLEF 202
MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification
Recent research in disaster informatics demonstrates a practical and
important use case of artificial intelligence to save human lives and suffering
during natural disasters based on social media contents (text and images).
While notable progress has been made using texts, research on exploiting the
images remains relatively under-explored. To advance image-based approaches, we
propose MEDIC (Available at: https://crisisnlp.qcri.org/medic/index.html),
which is the largest social media image classification dataset for humanitarian
response consisting of 71,198 images to address four different tasks in a
multi-task learning setup. This is the first dataset of its kind: social media
images, disaster response, and multi-task learning research. An important
property of this dataset is its high potential to facilitate research on
multi-task learning, which recently receives much interest from the machine
learning community and has shown remarkable results in terms of memory,
inference speed, performance, and generalization capability. Therefore, the
proposed dataset is an important resource for advancing image-based disaster
management and multi-task machine learning research. We experiment with
different deep learning architectures and report promising results, which are
above the majority baselines for all tasks. Along with the dataset, we also
release all relevant scripts (https://github.com/firojalam/medic).Comment: Multi-task Learning, Social media images, Image Classification,
Natural disasters, Crisis Informatics, Deep learning, Datase
Zero- and Few-Shot Prompting with LLMs: A Comparative Study with Fine-tuned Models for Bangla Sentiment Analysis
The rapid expansion of the digital world has propelled sentiment analysis
into a critical tool across diverse sectors such as marketing, politics,
customer service, and healthcare. While there have been significant
advancements in sentiment analysis for widely spoken languages, low-resource
languages, such as Bangla, remain largely under-researched due to resource
constraints. Furthermore, the recent unprecedented performance of Large
Language Models (LLMs) in various applications highlights the need to evaluate
them in the context of low-resource languages. In this study, we present a
sizeable manually annotated dataset encompassing 33,605 Bangla news tweets and
Facebook comments. We also investigate zero- and few-shot in-context learning
with several language models, including Flan-T5, GPT-4, and Bloomz, offering a
comparative analysis against fine-tuned models. Our findings suggest that
monolingual transformer-based models consistently outperform other models, even
in zero and few-shot scenarios. To foster continued exploration, we intend to
make this dataset and our research tools publicly available to the broader
research community. In the spirit of further research, we plan to make this
dataset and our experimental resources publicly accessible to the wider
research community.Comment: Zero-Shot Prompting, Few-Shot Prompting, LLMs, Comparative Study,
Fine-tuned Models, Bangla, Sentiment Analysi
MEDIC: a multi-task learning dataset for disaster image classification
Recent research in disaster informatics demonstrates a practical and important use case of artificial intelligence to save human lives and suffering during natural disasters based on social media contents (text and images). While notable progress has been made using texts, research on exploiting the images remains relatively under-explored. To advance image-based approaches, we propose MEDIC (https://crisisnlp.qcri.org/medic/index.html), which is the largest social media image classification dataset for humanitarian response consisting of 71,198 images to address four different tasks in a multi-task learning setup. This is the first dataset of its kind: social media images, disaster response, and multi-task learning research. An important property of this dataset is its high potential to facilitate research on multi-task learning, which recently receives much interest from the machine learning community and has shown remarkable results in terms of memory, inference speed, performance, and generalization capability. Therefore, the proposed dataset is an important resource for advancing image-based disaster management and multi-task machine learning research. We experiment with different deep learning architectures and report promising results, which are above the majority baselines for all tasks. Along with the dataset, we also release all relevant scripts (https://github.com/firojalam/medic).Other Information Published in: Neural Computing and Applications License: https://creativecommons.org/licenses/by/4.0See article on publisher's website: http://dx.doi.org/10.1007/s00521-022-07717-0</p